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Can Using ACT Online Prep Improve Score Gains? 
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preparation programs. 


Introduction 


Test preparation plays an important role in high-stakes standardized testing. While test preparation 
companies may claim large, at times unrealistic, gains associated with product use, much of which can 
be very costly, scientific research supports a more moderate impact of test preparation (Briggs, 2009; 
Montgomery & Lilly, 2012; Powers, 1993). These impacts should not, however, be understated as 
even small improvements in test scores can make a difference for college admissions and 

scholarship eligibility. 


ACT offers a number of test preparation opportunities for students who are planning to take the ACT® test.' 
ACT’s test preparation suite of solutions includes the Official ACT Prep Guide, ACT® Academy™, ACT 
Online Prep (AOP), ACT Rapid Review Live, ACT Rapid Review On Demand, and ACT Rapid Review All 
Access. Each of these solutions caters to different student learning styles and strategies. 


In this study, we focus on student’s usage of AOP. AOP provides two learning paths for students. The 
first is structured, comprehensive coverage of all subjects at all levels regardless of student's starting 
point. The second is an adaptive learning plan based on input from diagnostic tests to identify areas of 
weakness to target first. AOP presents official ACT test materials in a format that ensures students can 
cover all content areas tested. In particular, it affords access to over 200 hours of content including over 
2,400 practice items. 


The current study sought to examine the score gains students have attained between official ACT test 
administrations when using AOP between the tests. 


Analytical Sample 

For the present analysis, a sample was constructed which included ACT-tested students who had: 
e enrolled in AOP during high school as a junior or senior, 
e taken an ACT test within the six months prior to getting access to AOP, and 


e taken a subsequent ACT test after gaining access to AOP that was closest to, but no more than 
three months after, the end of their access period. 


These time bounds allowed for an examination of ACT score gains, with AOP usage occurring between 
tests. This analysis included all ACT retested AOP students since December 2015 and was limited to 
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students who had valid data for all student characteristics examined. This resulted in a sample of 46,394 
students. The sample had a slightly greater number of female students than male students, most of the 
students were Caucasian, and most students came from a family with an income greater than $60,000. 
Descriptive statistics for the sample are provided in Table 1. 


Table 1. Sample Characteristics 


%/Mean N 


Gender (%) 
Female 54.5 25,269 
Male 45.5 21,125 
Race/Ethnicity (%) 
African American to 3,472 
American Indian 0.5 217 
White 71.4 33,134 
Hispanic 8.4 3,876 
Asian 4.3 1,986 
Native Hawaiian 02 71 
Two or More Races 4.3 2,003 
Prefer Not to Respond 3.5 1,635 
Family Income (%) 
<$24,000 46 2,113 
$24 ,000-$36,000 5.0 2,305 
$36,000-$50,000 6.4 2,954 
$50,000-$60,000 6.3 2,908 
$60,000-$80,000 2 5,180 
$80,000-$100,000 14.1 6,549 
$100,000-$120,000 14.6 6,780 
$120,000-$150,000 12.5 5,818 
$150,000-$200,000 25.4 11,787 
Expected Educational Level (%) 
Voc-Tech Program 0.4 173 
Associate’s Degree 0.9 395 
Bachelor’s Degree 46.6 21,621 
1-2 Years of Graduate Program 22.8 10,582 
Doctorate or Professional Degree 28.9 13,410 
Other 0.5 213 
Highest Parental Education (%) 
Less than High School 1.0 446 
High School Graduate/GED 5:3 2,454 
Business/Technical School or Certificate Program 2.3 1,058 
Some College, No Degree or Certificate 1.9 3,470 
Associate's Degree 8.2 3,780 
Bachelor's Degree 30.0 17,156 
1-2 Years of Graduate Program 26.2 12,162 
Doctorate or Professional Degree 12.7 5,868 
Needs Help in Educational Skills or Occupational planning (%) 
Yes 60.1 21,807 


No 39.9 18,517 
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Table 1. Continued 


Testing with a Fee Waiver (%) 


Yes 3.9 1,814 
No 96.1 44,580 
Grade Level (%) 
11 274 12,566 
12 72.9 33,828 
Taken Mathematics Beyond Algebra II (%) 
Yes 64.5 29,936 
No 35.5 16,458 
Taken Biology, Chemistry, and Physics (%) 
Yes 36.9 17,104 
No 63.1 29,290 
Mean High School GPA 3:52 46,394 
Mean Pre-ACT Composite Score 22.14 46,394 
Mean Post-ACT Composite Score 23.39 46,394 
Mean ACT Composite Score Gain 1.25 46,394 
Mean Months Elapsed Between Pre- and Post-ACT Tests 5.83 46,394 
Number of Times Taken the ACT 3.39 46,394 


Statistical Analysis 


The association between AOP usage and ACT Composite score gain was evaluated using Analysis of 
Covariance (ANCOVA) methods. This analytical methodology allows us to test the difference in group 
means associated with AOP usage while controlling for all control and usage variables simultaneously. This 
allows the exploration of the incremental contribution to ACT score for each of the included usage variables 
while controlling for student characteristics. While this methodology does not afford one to make causal 
conclusions, the ability to control for student characteristics provides a more accurate measure of the 
association between AOP usage and ACT Composite score gain. 


The present analysis accounted for 13 student characteristics. These include gender, student’s grade level, 
race/ethnicity, student's expected educational attainment, family income, highest parental education, 
whether a student tested with a fee waiver, having taken mathematics courses beyond Algebra II, having 
taken at least biology, chemistry, and physics, high school GPA, prior ACT Composite score, the number of 
months elapsed between their pre- and post-ACT tests, and the total number of times they had taken the 
ACT. Each of these 13 covariates were included because they have the potential to either directly or 
indirectly impact how a student may perform on the ACT. 


To gain a better understanding of the gain in ACT Composite score associated with particular features of 
AOP, this study also looked at the activity of students in the platform (see Table 2). There were five AOP 
usage statistics that were investigated: 


1. The number of days a student was active on AOP. This included any and all activity on the system. 


2. The number of content lessons reviewed. These lessons focus on content reinforcement in English, 
mathematics, reading, and science. 
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3. The number of practice sessions a student took. These practice sessions provide students with 
practice items on each section of content covered by the ACT. 


4. The number of full-length practice tests taken by a student. This timed practice test replicates the 
experience of taking the ACT by using retired ACT test forms. 


5. The number of system resets performed by the student. Students can reset parts of the AOP 
system or the entire platform. This affords the students the opportunity to erase current work done 
on the system, such as practice test responses, and start as if new to the system. 


For each of the AOP usage variables considered, the distribution was examined to establish levels of each 
variable. In some cases, quartile values were used to guide level creation. For other variables, it made 
more sense to decompose larger intervals that resulted from the quartiles. In these cases, judgments were 
made based on the distribution, interval size, and an understanding of each variable. The categorization of 
each AOP usage variable is provided in Table 2. 


Table 2. AOP Usage Characteristics 


Number of Days Using AOP (%) 


0 39.6 18,365 
1 10.4 4,843 
2-5 25.8 11,989 
6-10 13.1 6,069 
11-20 Tal 3/053: 
21+ 3.4 1,575 
Number of AOP Lessons Reviewed (%) 
0 45.1 20,912 
1-14 12.6 5,843 
15-27 14.4 6,695 
28 - 54 14.0 6,510 
55+ 13.9 6,434 
Number of AOP Practice Sessions (%) 
0 59.7 - 
1 11.8 6,069 
2-5 17.8 6,510 
6-10 7.0 6,434 
11-15 2.5 20,912 
16+ AZ 6,695 
Number of Full Length Practice Tests (%) 
0 95.6 44,342 
1 3.3 1,537 
2+ V1 515 
Number of AOP System Resets (%) 
0 85.5 39,662 
1-5 12.4 5,761 
6-10 1.3 621 


11+ 0.8 350 
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Results 
Days Active in AOP 


As seen in Figure 1, this analysis found that, in general, ACT Composite score gain increased as the 
number of days active in the AOP system increased. Students who had at least 21 active days had a score 
gain much larger than that for students who had an account but never used AOP (1.59 versus 1.16). 


Figure 1. ACT Score Gain by the Number of Days Active in AOP 


ACT Composite 
Score Gain 


0 1 2-5 6-10 11-20 21+ 
Number of Days Using AOP 


Table 3 displays the test of mean differences between different days of AOP usage. This table shows 

that most of the pairwise combinations of the six categories of days showed significant differences. 
Statistically, there was no difference between the average gain score for students who had zero active days 
in AOP and students who had a single active day in AOP. Additionally, most score gain means associated 
with more than one active day were statistically greater than that seen for students with zero or one active 
day in AOP.? Moreover, the group means for students who had 2-5, 6-10, 11-20, and 20 or more days were 
all statistically different from each other. The effect sizes for these difference are small ($0.08). 


Table 3. Pairwise Tests of Mean Differences Between Days Active in AOP 


Days Mean Difference Standard Error Pr > |t| Std Dev Effect Size 
1 0 0.04 0.05 0.36 5.80 0.01 
1 2-5 -0.07 0.04 0.06 5.80 0.01 
1 6-10 -0.13 0.05 0.00 5.80 0.02 
1 11-20 -0.25 0.06 <.0001 5.80 0.04 
1 21+ -0.40 0.08 <.0001 5.80 0.07 
2-5 0 0.11 0.05 0.02 5.80 0.02 
2-5 6-10 -0.07 0.03 0.05 5.80 0.01 
2-5 2 -0.33 0.07 <.0001 5.80 0.06 
6-10 0 0.18 0.06 0.00 5.80 0.03 
11-20 0 0.29 0.07 <.0001 5.80 0.05 
11-20 2-5 0.18 0.05 0.00 5.80 0.03 
11-20 6-10 0.12 0.04 0.01 5.80 0.02 
11-20 21+ -0.15 0.06 0.02 5.80 0.03 
21% 0 0.44 0.09 <.0001 5.80 0.08 
21 6-10 0.26 0.07 <.0001 5.80 0.05 


Note: Effect size was calculated as the difference between means for greater number of active days and lesser 
number of active days, divided by the 2018 population ACT Composite score standard deviation. Differences in this 
table may not match differences conveyed in the histogram due to rounding. 
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Number of Lessons Reviewed 


Average ACT Composite score gain appears to have a fairly flat relationship with the number of lessons 
reviewed (see Figure 2). In this analysis, students who reviewed 1-14 or 15-27 lessons had a slightly lower 
average score gain than students who reviewed no lessons or more than 27 lessons. Students who 
reviewed at least 55 lessons had a mean score gain range of 1.33 points. 


Figure 2. ACT Score Gain by the Number of Lessons Reviewed 


1.2 
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ACT Composite 
Score Gain 


0 1-14 15-27 28 - 54 55+ 
Number of AOP Lessons Reviewed 


Table 4 shows the tests of significance for the different levels of number of lessons taken. We can see that 
students who took between 1 and 27 lessons had a lower gain score than students who reviewed no 
lessons or more that 27 lessons. The associated effect sizes were at most 0.03. 


Table 4. Pairwise Tests of Mean Differences Between Numbers of Lessons Taken in AOP 


Number of Number of Mean Standard 

| =X-X-Y0] at) | =X-X-Yo) ah Difference Error ad ee || Std Dev Effect Size 
<=14 0 -0.11 0.05 0.02 5.80 -0.02 
<=14 15 - 27 -0.01 0.03 0.86 5.80 0.00 
<=14 28 - 54 -0.14 0.04 0.00 5.80 0.02 
<=14 55+ -0.17 0.04 <.0001 5.80 0.03 
15 - 27 0 -0.10 0.05 0.03 5.80 0.02 
15-27 28 - 54 -0.13 0.04 0.00 5.80 0.02 
15 - 27 Dot -0.17 0.04 <.0001 5.80 0.03 
28 - 54 0 0.03 0.05 0.55 5.80 0.00 
28 - 54 55+ -0.04 0.04 0.35 5.80 0.01 
55+ 0 0.06 0.05 0.21 5.80 0.01 


Note: Effect size was calculated as the difference between means for greater number of lessons taken and lesser 
number of lessons taken, divided by the 2018 population ACT Composite score standard deviation. Differences in 
this table may not match differences conveyed in the histogram due to rounding. 


Number of Practice Sessions 


ACT Composite score gains appear larger as students take more practice sessions (see Figure 3). For 
example, students who took 0, 6 to 10, and at least 16 practice session had an average ACT Composite 
score gain of 1.22, 1.39 and 1.60. There appears to be a sharp increase in the gain score associated with 
completing more than one practice session. The average score gain estimate amongst the students most 
active in practice sessions, at least 16 sessions, was 1.60. 
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Figure 3. ACT Score Gain by the Number of Practice Sessions Taken 
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Table 5 shows that, generally speaking, the difference in gain scores observed as the number of lessons 
increases are statistically significant. These differences are less pronounced for students taking over six 
practice sessions. All effect sizes are at most 0.07. 


Table 5. Pairwise Tests of Mean Differences Between Numbers of Practice Sessions Taken in AOP 


Practice Practice Mean Siete ne| 

SY=t-¥-4 [0] al] SY=t-¥-9 [0] a] Difference Error Pr > |t| Std Dev Effect Size 
1 0 -0.04 0.03 0.31 5.80 -0.01 
1 2-5 -0.09 0.04 0.01 5.80 0.02 
1 6-10 -0.20 0.05 <.0001 5.80 0.04 
1 11-15 -0.27 0.07 0.00 5.80 0.05 
1 16+ -0.41 0.10 <.0001 5.80 0.07 
2-5 0 0.06 0.03 0.08 5.80 0.01 
2-5 6-10 -0.11 0.04 0.01 5.80 0.02 
6-10 0 O17 0.05 0.00 5.80 0.03 
11-15 0 0.23 0.07 0.00 5.80 0.04 
11-15 2-5 0.17 0.07 0.01 5.80 0.03 
11-15 6-10 0.06 0.07 0.36 5.80 0.01 
11-15 16+ -0.15 0.10 0.15 5.80 0.03 
16+ 0 0.38 0.10 0.00 5.80 0.07 
16+ 2-5 0.32 0.10 0.00 5.80 0.06 
16+ 6-10 0.21 0.10 0.03 5.80 0.04 


Note: Effect size was calculated as the difference between means for greater number of practice sessions and 
lesser number of practice sessions, divided by the 2018 population ACT Composite score standard deviation. 
Differences in this table may not match differences conveyed in the histogram due to rounding. 
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Number of Practice Tests 


AOP offers students the ability to practice taking the exam with two full length, retired ACT practice tests. 

Students have an unlimited number of times to practice using these tests. The score gain associated with 
taking practice tests increases as more practice tests are taken. In fact, for students who take at least two 
full-length practice tests, the associated mean gain is 1.69 points. 


Figure 4. ACT Score Gain by the Number of Full-Length Practice Tests Taken 
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Table 6 shows that taking at least one full length practice test was associated with a significantly greater 
ACT score gain than taking none, while the difference between taking one and two or more practice tests 
was not significant. The estimated effect size for taking at least two practice tests is 0.08. 


Table 6. Pairwise Tests of Mean Differences Between Numbers of Full-Length Practice Tests Taken 


Number of Long Number ofLong Mean Sie lalek-lne| 

Form Tests Form Tests Difference Error Pr > |t| Std Dev Effect Size 
1 0 0.28 0.05 <.0001 5.80 0.05 
1 2+ -0.17 0.10 0.08 5.80 0.03 
2+ 0 0.45 0.09 <.0001 5.80 0.08 


Note: Effect size was calculated as the difference between means for greater number of practice tests and lesser 
number of practice tests, divided by the 2018 population ACT Composite score standard deviation. Differences in 
this table may not match differences conveyed in histogram due to rounding. 


Number of System Resets 


When a student resets the entire AOP system, their prior work on the platform is erased, and they can 
retake all components of the system. Resetting the system allows students to revisit practice sessions and 
practice tests without prior scores or responses influencing their progress. There appears to be a positive 
association with resetting the system between 6 and 10 times and students’ ACT gain scores. Students 
with 6 to 10 resets had the highest average gain score of 1.41 points. 


© 
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Figure 5. ACT Score Gain by the Number of AOP System Resets 
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The pairwise tests of mean differences show that the gain score for students with 6-10 resets was higher 
than students who never reset AOP and similar to the gain score for students who reset AOP at least once 
or more than 10 times. Relevant effect sizes were at most 0.03. 


Table 7. Pairwise Tests of Mean Differences Between Numbers of AOP Resets 


Number of Number of Mean Standard 

Resets Resets Difference Error ad ee || Std Dev Effect Size 
1-5 0 0.04 0.03 0.21 5.80 0.01 
1-5 6-10 -0.13 0.08 0.10 5.80 0.02 
1-5 V+ -0.05 0.11 0.66 5.80 0.01 
6-10 0 0.17 0.08 0.03 5.80 0.03 
11+ 0 0.09 0.11 0.43 5.80 0.01 
ilies 6-10 -0.09 0.13 0.50 5.80 -0.02 


Note: Effect size was calculated as the difference between means for greater number of resets and lesser number 
of resets, divided by the 2018 population ACT Composite score standard deviation. Differences in this table may 
not match differences conveyed in histogram due to rounding. 


Conclusion 


This study highlighted the positive association between using ACT Online Prep and ACT score gains over 
time. In particular, this study demonstrates the positive association with greater numbers of active days in 
AOP, the number of practice sessions done, the number of full-length practice tests completed, and the 
number of system resets. The association of each AOP usage variable with ACT score gain was estimated 
using a multiple linear regression model that controlled for several student characteristics as well as all 
other AOP usage variables. The results provide an estimate of the difference in ACT score gain between 
not using AOP and using it “optimally.” For example, we can estimate that if a students used AOP for over 
21 days, reviewed over 55 lessons, took at least 16 practice sessions and at least two practice tests, and 
reset the system between 6-10 times they would have an average gain score that would be 1.51 points 
higher than if they did not use the AOP system. 


Students who purchased AOP, yet did not use the product, also saw gains in their ACT retest score, as 
would be expected with the additional instruction that occurs between tests. The average gain for students 
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who did not use AOP was 1.13, while the average gain for students who use AOP optimally is 2.64 (i.e., 
1.13 + 1.51). 


There are certain limitations to this study that should be borne in mind. The most important is that this 
analytical method only allows us to speak of the association between gain scores and using features of 
AOP. Based on this study, however, causal claims cannot be made. Rather, this study helps us to 
understand the experience students are having with their AOP usage. 


A second limitation is that information on additional test preparation activities students may have been 
involved with were not available. It is possible that AOP was the sole or primary activity performed between 
the two test dates, but it is also possible that it served as a complimentary activity to other test preparation 
activities. If students who did not use AOP were more likely to engage in other types of test preparation not 
captured by this study, the study likely underestimates the impact of AOP usage. An attempt to capture 
some of this variability in other test prep activities was made by including the number of months elapsed 
between the two tests in our model. It was thought that as the number of months between the two tests 
increased, the opportunity for other activities, including school learning, increased. 


Another point to consider was that by using an ANCOVA approach that included all types of usage 
simultaneously, we see the incremental effects of each, but these estimates are likely smaller than if a 
model was estimated with only one type at a time. Notwithstanding these limitations, this study 
demonstrates the positive association between AOP usage and ACT Composite score gains over time. 


Notes 


1. In 2017, over two million high school seniors took the ACT. 
The mean difference between students with 2-5 days of activity was significantly different than that 
of students with zero days of activity but was not significantly different than students with one day 
of activity. 
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